Papers with push-pull formulation
Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking (2026.acl-long)
Copied to clipboard
| Challenge: | Existing attacks optimize image perturbations to maximize harmful output likelihood, but suffer from slow convergence due to gradient conflict between adversarial objectives and the model’s safety-retrieval mechanism. |
| Approach: | They propose a push-pull approach which suppresses attention to system-prompt tokens and anchors generation on adversarial image features to avoid collisions. |
| Outcome: | The proposed approach reduces gradient conflict by 45% and achieves 94.4% attack success rate on Qwen-VL (vs. 68.8% baseline) with 40% fewer iterations. |